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1.  Introduction 


The  Computational  Science  Environment  (CSE)  provides  users  with  a  standard  development 
environment  as  well  as  open  source  software  tools  and  utilities  for  data  analysis  and 
visualization.  A  significant  advantage  of  using  CSE  is  its  ability  to  provide  researchers  and 
engineers  with  a  development  foundation  capable  of  supporting  their  software  project.  This  is 
supported  through  a  CSE  subproject  or  “add-on.”  CSE  provides  a  straightforward  framework  for 
creating  and  automatically  testing  add-ons  along  with  the  ability  to  customize  the  build  process 
by  specifying  particular  compilers,  optimization  flags,  and  external  libraries  to  meet  the 
researcher’s  needs.  CSE  add-ons  are  a  simple  way  for  researchers  and  developers  to  extend  the 
common  baseline  set  of  tools,  libraries,  and  applications  in  CSE.  The  CSE  team  has  worked 
closely  with  the  Multiple  Object  Evolutionary  Strategies  (MOES)  developers  as  well  as  the 
Multiscale  Reactive  Modeling’s  (MSRM)  Infrastructure  team  to  design  a  modular  build  and  test 
system  that  simplifies  the  entire  build  process  and  make  the  projects  easier  to  maintain  and 
modify. 


2.  Configuration  Management 


2.1  The  Software  Repository 

Source  code  management  for  the  MSRM  project  is  done  through  Git.  Git  is  a  distributed  version 
control  system  unlike  Subversion  (SVN),  which  is  a  centralized  version  control  system.  Because 
Git  is  a  distributed  version  control  system,  when  a  user  performs  a  checkout  (clone),  every 
revision  of  every  file  is  available  as  part  of  the  local  copy.  An  additional  benefit  of  Git  is 
branching.  While  branching  is  not  unique  to  Git,  one  benefit  is  being  able  to  completely  remove 
a  branch  (there  is  no  trace  like  there  is  in  SVN),  so  branches  can  be  easily  created  and  discarded 
as  necessary. 

Git  Repository  permissions  are  handled  at  the  operating  system  (OS)  and  filesystem  level 
through  the  use  of  Linux  groups  and  Filesystem  action  control  lists  (ACLs)*.  Each  project  is 
designated  a  unique  group  name  for  the  facilitation  of  permissions,  only  individuals  designated 
by  the  project  point  of  contact  (POC)  will  be  added  to  the  group. 


The  ACL  is  a  list  of  permissions  attached  to  an  object.  This  list  specifies  which  users  or  groups  are  granted  permissions  to  an 
object  and  what  actions  can  be  performed  on  that  object. 
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The  following  commands  are  used  to  provide  a  specific  group  with  read  and  write  permission  to 
their  repository: 

setfacl  -R  -m  g:cse:rwX  cse_stable 

find  repo  -type  d  I  xargs  setfacl  -R  -m  d:g:cse:rwX 

There  are  two  software  repositories  for  each  Atomistic  add-on.  Each  add-on  has  a  development 
repository  and  a  production  repository.  All  developers  have  read  and  write  permissions  to  the 
development  repository.  Only  individuals  deploying  the  add-on  have  full  access  to  the 
production  repository.  Each  of  these  repositories  is  built,  tested,  and  reported  on  nightly.  The 
production  repository  is  built,  tested,  and  deployed  on  various  high  performance  computing 
(HPC)  systems  as  directed. 

2.2  Software  Configuration  Management 

Because  the  team  is  “geographically  distributed,”  an  online  interface  was  necessary  to  simplify 
communication.  The  team  uses  Redminet,  an  open-source,  database-driven  project  management 
Web  application.  The  Redmine  application  offers  a  substantial  preconfigured  solution  that  is 
easy  to  setup  and  can  be  customized  to  better  fit  a  particular  project.  Redmine  was  chosen  as  it 
provides  a  number  of  features  that  are  of  interest  for  team  interaction.  The  most  significant 
features  were  the  ability  to  support  multiple  projects,  a  customizable  issue  tracking  system, 
e-mail  and  news  notifications,  Wiki  support,  access  control,  and  integration  with  several  source 
code  management  systems  (e.g.,  Git,  as  described  previously).  Redmine’s  tracking  system 
provides  a  simple  solution  for  managing  bugs,  general  support  issues,  and  requests  for  new 
features  allowing  the  entire  team  to  determine  priorities,  provide  updates,  and  know  the  status  of 
any  outstanding  items.  As  items  in  the  tracking  system  are  updated,  the  responsible  personnel 
and  others  with  an  interest  are  informed  via  e-mail.  Redmine  also  provides  a  common  place  for 
current  source  code  via  the  connected  Git  repository,  as  well  as  code  and  user  documentation  via 
its  Wiki  and  file  storage  features.  Collectively,  these  features  reduce  the  time  needed  to  oversee 
and  manage  multiple  projects,  allow  the  team  to  keep  track  of  issues  specific  to  their  assigned 
projects,  and  provide  a  common  place  for  project  related  communication. 


3.  CMake  and  Modules 


The  CSE  MOES  add-on  uses  CMake  to  drive  build  process.  CMake  is  an  open-source  cross 
platform  build  system  with  strong  system  introspection  capabilities.  There  are  numerous  macros 
distributed  with  CMake  to  discover  what  packages  are  installed  on  a  system,  their  version 
numbers,  and  where  these  packages  are  installed.  Because  the  MOES  code  was  needed  by  the 
team’s  scientists  on  several  different  HPC  architectures,  the  system  introspection  capabilities  of 


f  Redmine  Web  site,  http://www.redmine.org  (accessed  2011). 
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CMake  significantly  simplified  deployment.  There  is  a  top  level  CMakeLists.txt  file  that 
includes  all  of  the  sub-packages  as  well  as  the  module  installation  code. 

The  following  is  from  the  top  level  CMakeLists.txt  project  it  sets  up  the  default  installation 
prefix  and  includes  all  of  the  sub-package  directories.  The  CSE  provides  template  files  for  the 
top  level  CMakeLists.txt  file: 

#  Add-on  CSEADDONCVS  Packages 
cmake_minimum_required(VERSION  2.8) 

#  The  name  of  the  build  project 
project(CSEADDONMOES) 
set( SUBPROJECT  “moes”) 
set(HOME_DIR  MOES) 
include(CTest) 

#  Enable  testing  for  the  project 
ENABLE_TESTING() 

include(ExternalProj  ect) 

set(base  “${CMAKE_BINARY_DIR}/CMakeExternals”) 
set_property(DIRECTORY  PROPERTY  EP  BASE  ${base}) 

#  Find  the  CSE  installation  on  this  system 

find_path(CSE_HOME  Release  $ENV{CSE_HOME}  /usr/cta/CSE  $ENV{HOME}/CSE) 
list( APPEND  CMAKE  MODULE  PATH  “${CSE_HOME}/Misc/CMake”) 
find_package(C  SE) 

#  Default  installation  prefix  is  /home/userid,  this  can  be  changed  at  configure  time 

#  To  change  the  Install  prefix  cmake  -DMOES_INSTALL_PREFIX=/prefix/you/want 

set  (MOES  INSTALL  PREFIX  “$ENV{HOME}/${HOME_DIR}”  CACHE  PATH  “Install 
Path”) 

set  (CMAKE  INSTALL  PREFIX  ${MOES_INSTALL_PREFIX}  CACHE  INTERNAL  ““ 
FORCE) 

find_program(PATCH_PROGRAM  patch 
PATHS  /usr/bin) 

#  include  all  subdirectories 
addsubdirectory(airebo) 
addsubdirectory(lpsolve) 
addsubdirectory(reac) 
addsubdirectory(moes) 
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Modules  are  automatically  built  and  installed  with  each  CSE  add-on.  Modules  are  a  command¬ 
line  tool  providing  dynamic  modification  of  a  user’s  environment,  thus  eliminating  the  need  for 
the  user  to  modify  their  own  environment  variables  (PATH,  LDLIBRARYPATH, 
PYTHONPATH)  to  compile,  test,  and  run  code.  The  CSE  team  provides  a  default  file  for  each 
module  that  is  populated  by  package  specific  variable  names  at  configure  time. 


The  MOES  project  installs  the  following  modules  for  use  by  MOES  developers  and  individuals 
perfonning  batch  runs  using  the  MOES  code: 


- /usr/cta/CSE. atomistic. 201 1-04-22/MOES/modules - 

cse-msrm-atomistic/airebo/1 . 1 

cse-msrm-atomistic/moes/1 .0 

cse-msrm-atomistic/airebo/latest 

cse-msrm-atomistic/moes/latest 

cse-msrm-atomistic/lp  solve/ 5 . 5 

cse-msrm-atomistic/reac/ 1 .2 

cse-msrm-atomistic/lp  solve/latest 

cse-msnn-atomistic/reac/latest 

4.  Automated  Build  and  Test 


CTest  is  a  testing  tool  distributed  with  CMake.  This  tool  can  be  used  to  automate  building  and 
testing  of  a  project.  CTest  can  also  submit  testing  results  to  a  dashboard  for  display  and  review. 
CDash  is  a  product  distributed  with  Kitware’s  CMake.  CDash  is  a  Web-based,  open-source 
software  testing  server.  CDash  organizes  and  displays  testing  results  on  a  simple,  easy  to 
understand  Web  page.  The  immediate  feedback  that  developers  receive  on  the  dashboard  helps 
to  encourage  careful  testing  and  code  review  prior  to  submitting  code  modifications  to  the 
repository. 

In  order  to  submit  to  a  local  dashboard  it  is  necessary  to  have  a  CTestConfig.cmake  file.  The 
following  is  a  basic  sample  of  a  CTestConfig.cmake  file: 

set(CTEST_PROJECT_NAME  “MOES  Development”) 
set(CTEST_NIGHTLY_START_TIME  “21:00:00  EDT”) 

#use  https 

set(CTEST_DROP_METHOD  “https”) 
set(CTEST_DROP_SITE  “your .web. dash.site”) 

set(CTEST_DROP_LOCATION  “/CDash/submit.php?project=MOES_Development”) 

set(CTEST_DROP_SITE_CDASH  TRUE) 

set(CTEST_CURL_OPTIONS 

“CURLOPT_SSL_VERIFYPEER_OFF;CURLOPT_SSL_VERIFYHOST_OFF”) 


4 


For  the  CSE  MOES  add-on,  each  package  is  tested  individually  and  the  entire  project  is  also 
tested  as  a  whole.  Because  all  of  the  packages  in  MOES  use  CMake  to  build,  all  of  the 
regression  tests  leverage  CTest.  The  CTest  output  is  stored  locally  in  an  extensible  markup 
language  (XML)  fde  that  gets  submitted  to  the  CDash  quality  assurance  dashboard  upon 
completion  of  the  builds  and  tests.  In  order  to  ensure  that  software  modifications  did  not  impact 
numerical  accuracy,  test  results  are  automatically  validated  against  a  known  set  of  values.  This 
verification  is  done  through  a  bash  script  that  gets  executed  by  CTest;  value  tolerances  can  be  set 
in  this  script  if  necessary.  The  following  line  needs  to  be  added  to  the  CMakeLists.txt  file  enable 
testing  for  the  project: 

enable_testing() 

The  CMake  tests  are  added  to  the  build  simply  by  adding  the  following  lines  to  the 
CMakeLists.txt  file,  these  lines  call  the  bash  scripts  that  execute  the  tests: 
add_test(TestMoesCalculate  test_moes.sh) 
add_test(TestAireboDriver  test  airebo .  sh) 

Figure  1  shows  the  quality  assurance  dashboard  reports  the  results  of  the  test. 


Figure  1.  The  quality  assurance  dashboard  reports  the  results  of  the  test. 
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5.  Data  Analysis  and  Visualization 


Using  several  of  the  tools  available  in  CSE  (Python,  NumPy,  PyQt,  and  Matplotlib),  the  CSE 
team  worked  with  the  MSRM  Infrastructure  team  and  the  MOES  developers  to  assist  with  the 
creation  of  data  conversion  and  visualization  programs.  A  Python  program  was  developed  to 
translate  the  MOES  output  code  into  a  CSV  file  that  can  be  used  by  numerous  spreadsheet  and 
graphing  packages.  A  Python  program  was  also  developed  to  automatically  generate  graphs  of 
the  data  output  using  Python,  NumPy,  and  Matplotlib. 

Figures  2  and  3  shows  example  plots  of  the  MOES  data. 


Figure  2.  Example  plot  of  MOES  data. 


Figure  3.  Example  plot  of  MOES  data,  zoomed  to  show  more  curve  details. 
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6.  Conclusion 


CSE  offers  a  comprehensive  software  environment  that  is  flexible,  provides  modem  software 
application  program  interfaces  (APIs),  and  supports  development  efforts  scaling  from  individuals 
to  large  geographically  dispersed  teams.  An  integral  part  of  the  CSE  is  the  automated  building, 
testing,  and  reporting  system  for  all  software  development  projects.  The  CSE  “add-on” 
framework  allows  users  and  developers  to  create  domain-specific  capabilities  outside  of  the 
system-level  architecture. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ACLs 

action  control  lists 

APIs 

application  program  interfaces 

CSE 

Computational  Science 

HPC 

high  performance  computing 

HPCMP 

High  Performance  Computing  Modernization  Program 

MOES 

Multiple  Object  Evolutionary  Strategies 

MSRM 

Multiscale  Reactive  Modeling 

OS 

operating  system 

POC 

point  of  contact 

SVN 

Subversion 

XML 

extensible  markup  language 
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